Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Jul 4, 2005 11:32pm
Forum: movies Subject: Re: Search engine lagging behind

Here's one post about it; there are probably others:
http://www.archive.org/iathreads/post-view.php?id=25578

Reply to this post
Reply [edit]

Poster: Madhavananda Date: Jul 5, 2005 12:58am
Forum: movies Subject: Re: Search engine lagging behind

Tracey speaks about a lag of roughly three hours. However with my entries it's more like three hundred times three hours...

http://www.archive.org/search.php?query=creator%3A%22Madhavananda%20Das%22

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Jul 8, 2005 2:59am
Forum: movies Subject: Re: Search engine lagging behind

Saaaay... 8-)

I don't think I ever said our search engine download /batting average counts were typically 3 hours "stale" -- unless that was 8+ months ago?

In the past, we used to update the numbers in the SE (from DB data) every 3 hours (so you're on to something with the 3 hours... 8-)

However, we found the SEs were too fragile to handle this and were constantly blowing their pretty little heads up. So they just are lagging now.

Last night, I did a manual "refresh the batting averages and download counts" to both our SEs. In the very near future, we might switch to a single "live" SE, and do the "refresh the counts" every night and see how that goes...

As everything goes here, depends on priority... 8-)
--Tracey



Reply to this post
Reply [edit]

Poster: Madhavananda Date: Jul 8, 2005 3:17am
Forum: movies Subject: Re: Search engine lagging behind

I don't think I ever said our search engine download /batting average counts were typically 3 hours "stale" -- unless that was 8+ months ago?

Quoting from that thread linked to in this thread (Dec 17th):

Hi all, I'm Jon's replacement, Tracey...

Anyway, yes, we have 2-4 search engines running at any given time and they handling requests at random. Sometimes the search engines "drift" a little bit with respect to download counts and "batting average" information. They will typically resync up to be close in value roughly every 3 hours (at present).

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Jul 9, 2005 2:46am
Forum: movies Subject: Re: Search engine lagging behind

perfect!

and i see my sense of time wasn't too far off for once 8-)

so yah, after some "horror shows" of problems over the november and december holidays, we turned off the "refresh DL/BA counts from DB" features in the SEs, and dropped the 4 SEs to 2.

i had a fully rebuilt and reindexed single SE about 1-2 months ago with some expanded features all ready to go, but we never quite got the go ahead to push it live. so it just faded away.

the good news is that we can reindex one in about a day and flip to a single newer SE and turn on DB count refreshing again and see how it goes. hopefully sometime in august?

--tracey

Reply to this post
Reply [edit]

Poster: Madhavananda Date: Jul 9, 2005 3:40am
Forum: movies Subject: Re: Search engine lagging behind

One approach to balancing the load on the search engine would be to have the items report the new counts to the search engine index whenever the counts are incremented or changes are made. Then you would never have to "index" the search engine. :)

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Jul 9, 2005 3:48am
Forum: movies Subject: Re: Search engine lagging behind

yes, but there are soooooo many updates and soooo frequently that i think this would be prohibitive...

so we use the DB for that (it can handle that kind of update load), and then we designed the SE to take bulk total updates every 3 hours (which it couldn't handle reliably).

but good idea! 8-)

--t

Reply to this post
Reply [edit]

Poster: Madhavananda Date: Jul 9, 2005 3:53am
Forum: movies Subject: Re: Search engine lagging behind

Of course I haven't the singlest clue about your infrastructure... I'm thinking of load-balancing in terms of a single server here, and that probably isn't the environment you're in. In that case, having items calling across the network with each tiny update is of course a no-no.

The concept could be extended to engaging the items in "reporting" to the search engine index at set intervals, whatever the interval might be. Say, calling home every 20 or 50 downloads. Then you wouldn't have items reporting to the SE constantly.

This post was modified by Madhavananda on 2005-07-09 10:53:57

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Jul 10, 2005 4:02am
Forum: movies Subject: Re: Search engine lagging behind

Hi Madhavananda,

well, again, good ideas.
but the Major Issue we have with our Search Engine is that it's decently neglected and not squarely under someone's responsibility here.

So code changes to it, even little ones, take a long time and can be challenging.

So we're mostly trying to stick with whatever the SE has functionality-wise for now, and accommodate around it (since the rest of the backend is much more easily changeable).

Anyway, just thought a little more insight might help as you seem curious and helpful, as always!
8-)

--tracey

Reply to this post
Reply [edit]

Poster: Madhavananda Date: Jul 8, 2005 3:15am
Forum: movies Subject: Re: Search engine lagging behind

Hey, I was wondering what happened. Thanks! Maybe if you ran a manual refresh on a weekly basis or something... Then I wouldn't have to play with these ugly hacks that bring down my batting averages. :)